chore: split the output of SRF by max_block_size.
#13817
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
github-actions
bot
added
the
pr-chore
|
It's incorrect, we should implement it inside the operator, current impl does not reduce the memory. |
|
Ok, I misunderstood the requirement. |
RinChanNOWWW
force-pushed
the
issue-13815
branch
from
be9a22b to
0e0b8f2
Compare
RinChanNOWWW
changed the title
BlockOperator::FlatMap.max_block_size.
Output blocks by determined batch size.
RinChanNOWWW
force-pushed
the
issue-13815
branch
from
0e0b8f2 to
a38cdf1
Compare
RinChanNOWWW
force-pushed
the
issue-13815
branch
from
0398f6a to
8f6d534
Compare
Docker Image for PR
|
|
Great, large blocks in SRF now is working 👍 |
sundy-li
approved these changes
Nov 28, 2023
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
I hereby agree to the terms of the CLA available at: https://databend.rs/dev/policies/cla/
Summary
Add a new transform type
BlockingTransformto block the processing (do not pull new data from the input port) if it doesn't meet the requirement.We can use
BlockingTransformfor SRFs to reduce memory usage:If the output of a block after applying SRF (such as
unnest) is too large, we can only generate and output part of the result (bymax_block_size). It will not read new input data until the whole result is consumed by the downstream processor.This improvement makes the processor can release memory immediately when the data is consumed.
BlockOperator::FlatMapouput result by batches bymax_block_size#13815This change is